System and method for dynamic validation, correction of registration for surgical navigation

ABSTRACT

A system and method for dynamic validation, registration correction for surgical navigation during medical procedures involving confirmation of registration between previously registered virtual objects, in a common coordinate frame of a surgical navigation system and an operating room, and intra-operatively acquired imaging during the medical procedure in the common coordinate frame. The method involves displaying intra-operatively acquired imaging of the surgical field, containing the real objects corresponding to the previously registered virtual objects, with the real objects being tracked by a tracking system. The method involves overlaying a virtual image containing the previously registered virtual objects onto the intra-operatively acquired imaging, from the point of view of the intra-operatively acquired imaging, and detecting any misalignment between any the previously registered virtual objects contained in the virtual image and its corresponding real object contained in the intra-operatively acquired imaging.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a continuation application claiming the benefit of, and priorityto the following patent applications: U.S. patent application Ser. No.14/775,759, entitled “SYSTEM AND METHOD FOR DYNAMIC VALIDATION,CORRECTION OF REGISTRATION FOR SURGICAL NAVIGATION,” and filed on Sep.14, 2015; U.S. Provisional Application Ser. No. 61/799,735, entitled“SYSTEM AND METHOD FOR DYNAMIC VALIDATION AND CORRECTION OFREGISTRATION, AND RECOVERY OF LOST REFERENCE, FOR SURGICAL NAVIGATION,”and filed on Mar. 15, 2013; U.S. Provisional Application Ser. No.61/801,530, entitled “SYSTEMS, DEVICES AND METHODS FOR PLANNING,IMAGING, AND GUIDANCE OF MINIMALLY INVASIVE SURGICAL PROCEDURES, andfiled on Mar. 15, 2013; U.S. Provisional Application Ser. No.61/818,280, entitled “SYSTEMS, DEVICES AND METHODS FOR PLANNING,IMAGING, AND GUIDANCE OF MINIMALLY INVASIVE SURGICAL PROCEDURES,” andfiled on May 1, 2013; U.S. Provisional Application Ser. No. 61/800,155,entitled “PLANNING, NAVIGATION AND SIMULATION SYSTEMS AND METHODS FORMINIMALLY INVASIVE THERAPY,” and filed on Mar. 15, 2013; and U.S.Provisional Application Ser. No. 61/924,993, titled “PLANNING,NAVIGATION AND SIMULATION SYSTEMS AND METHODS FOR MINIMALLY INVASIVETHERAPY” and filed on Jan. 8, 2014, all of which are hereby incorporatedherein by reference in their entirety.

FIELD

The present disclosure relates to a system and method for dynamicvalidation and correction of registration, and recovery of lostreference, for surgical navigation during operations.

BACKGROUND

During a navigated surgical procedure, a surgeon typically needs tocorrelate the position of previously acquired imaging data (in threedimensions), that have been obtained from an imaging device or system(for example, ultrasound, CT or MRI data), with the physical position ofthe patient of whom is to be operated. In some systems, for a navigationprocedure a handheld instrument may be tracked using a tracking system;and a representation of the instrument's position and orientation may bedisplayed as an overlay on the three-dimensional imaging data from ascan of the patient's anatomy. To achieve this, a registration isobtained among the coordinate frame of the tracking system for thehandheld instrument, the physical location of the patient in physicalspace, and the coordinate frame of the corresponding image data of thepatient. Ensuring that the registration is aligned with and correspondsto the actual physical reality of the procedure is desirable andnecessary for maintaining surgeon confidence in the information beingpresented and in ensuring that the navigated procedure is accuratelyexecuted.

However, the registration tends to be difficult to measure; and itsaccuracy is difficult to quantify. In the related art, this accuracy hasbeen reported to a surgeon as a confidence or tolerance number at thetime that registration is computed. This was also described by thepaper, entitled “The Silent Loss of Navigation Accuracy;Research-Human-Clinical Studies,” Vol. 72, No. 5, May 2013, pages796-807. This number is not indicative of the complexity of registrationaccuracy, and, more significantly, is not indicative of the fact thataccuracy can vary in different parts of the surgical field. Further,this number is used as a one-time accept/reject criterion for theregistration. Once the registration is accepted typically, theregistration is assumed to be correct for the duration of the procedureor until the surgeon notices that something is significantly misaligned.

With the present state of the art, misalignment of the navigation systemis difficult to identify as a typical related art system only presents avirtual representation of the operating room (OR) procedure, and as suchit cannot be readily contrasted to the actual physical state of the ORat a given time. Currently, for a surgeon to measure registrationaccuracy during a procedure he or she typically locates the toolrelative to an identifiable location on the actual patient anatomy whilenoting the degree to which the location of the virtual tool is displacedfrom the same location relative to the virtualized patient anatomy,where such virtual tool is displayed as an overlay on thethree-dimensional imaging data from a scan of the patient's anatomy.Furthermore, once a registration misalignment is noticed, correcting forthe error tends to be difficult, and often not achievable. Additionally,non-uniform displacement of tissue during a procedure also tends to meanthat global corrections are not possible.

SUMMARY

The present disclosure involves a system and method for validatingregistrations, and detecting and correcting registration. An embodimentdisclosed herein provides a method of confirmation of correctregistration between one or more previously registered virtual objectsin a coordinate frame of a surgical navigation system (which is locatedin an operating room in which a medical procedure is to be performed)and intra-operatively acquired imaging during the medical procedure inthe coordinate frame of the surgical navigation system. Wherein apreviously registered virtual object may be a computed trackedinstrument visualization, or other computed tracked real objectvisualization. The surgical navigation system includes a trackingmechanism. The method includes displaying intra-operatively acquiredimaging of a surgical field containing one or more real objectscorresponding to the one or more virtual objects, the surgical fieldcontaining a pre-selected number of landmarks in fixed and knownlocations with respect to the one or more real objects, with thelandmarks being tracked by the tracking mechanism. The method includesoverlaying a virtual image (as generated by a virtual camera) containingthe one or more virtual objects previously registered onto theintra-operatively acquired imaging and detecting for any misalignment ornon-concordance between any one of the one or more previously registeredvirtual objects contained in the virtual image and its correspondingreal object contained in the intra-operatively acquired imaging, whereina presence of misalignment or any non-concordance is indicative of aregistration error.

An embodiment disclosed herein is a system for confirmation of correctregistration between one or more previously registered virtual objectsand intra-operatively acquired imaging during a medical procedure. Thesystem comprises a surgical navigation system having a coordinate frameof reference and including a tracking mechanism. The system furthercomprises a computer control system which includes a computer processor,and a storage device and a visual display both connected to the computerprocessor. The storage device has stored therein a visualization of oneor more previously registered virtual objects in the coordinate frame ofreference of the surgical navigation system. The system includes atleast one sensor for acquiring intra-operative imaging of a surgicalfield during the medical procedure in which the surgical field containsone or more real objects corresponding to the one or more virtualobjects and a pre-selected number of landmarks in known locations withrespect to the one or more real objects. The landmarks and the at leastone sensor are tracked by the tracking mechanism in the coordinate frameof the surgical navigation system. The computer processor is programmedwith instructions to receive and display the intra-operatively acquiredimaging of the surgical field containing the one or more real objectsand to overlay the virtual image containing the one or more virtualobjects previously registered onto the intra-operatively acquiredimaging and wherein any misalignment or non-concordance between any oneof the one or more previously registered virtual objects contained inthe virtual image and its corresponding real object contained in theintra-operatively acquired imaging is indicative of a registration errorbetween the virtual object and its corresponding real object in saidcoordinate frame of said surgical navigation system.

In an embodiment, the described system and methods can also providecorrections based on the difference between local tissue characteristicsand virtual instrument representations at the location that the surgeonis focusing on and a live video stream of the surgical field toimmediately visualize (and optionally automatically or manually correctfor) any difference between the expected (as calculated through aregistration) and actual positions of tracked instruments and imagedpatient tissue, which tends to achieve local, immediate, corrections ofregistration in an intuitive way.

In an embodiment, a method of confirmation of registration between oneor more previously registered virtual objects in a common coordinateframe of a surgical navigation system and an operating room in which amedical procedure is to be performed, and intra-operatively acquiredimaging during said medical procedure in said common coordinate frame,said surgical navigation system including a tracking system, the methodcomprises: displaying intra-operatively acquired imaging of a surgicalfield containing one or more real objects corresponding to said one ormore previously registered virtual objects, the real objects beingtracked by the tracking system; overlaying a virtual image containingthe one or more previously registered virtual objects onto theintra-operatively acquired imaging, from the point of view of theintra-operatively acquired imaging; and detecting for any misalignmentbetween any one of the one or more previously registered virtual objectscontained in the virtual image and its corresponding real objectcontained in the intra-operatively acquired imaging, wherein a presenceof misalignment is indicative of registration error between the virtualobject and its corresponding real object.

In yet another embodiment, a system for confirmation of correctregistration between one or more previously registered virtual objectsand intra-operatively acquired imaging during a medical procedure,comprises: a surgical navigation system having a coordinate frame andincluding a tracking mechanism; a computer control system including acomputer processor, a storage device and a visual display both connectedto said computer processor, said storage device having stored therein acomputed tracked instrument visualization of one or more previouslyregistered virtual objects in said coordinate frame of reference of saidsurgical navigation system; at least one sensor for acquiringintra-operative imaging of a surgical field during the medicalprocedure, said surgical field containing one or more real objectscorresponding to said one or more virtual objects; and said computerprocessor being programmed with instructions to receive and display saidintra-operatively acquired imaging of the surgical field and to overlaya virtual image from the point of view of the virtual camera onto theintra-operatively acquired imaging, wherein any misalignment between anyone of the one or more previously registered virtual objects containedin the virtual image and its corresponding real object contained in theintra-operatively acquired imaging is indicative of a registration errorbetween the virtual object and its corresponding real object.

A further understanding of the functional and advantageous aspects ofthe present disclosure can be realized by reference to the followingdetailed description and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments disclosed herein will be more fully understood from thefollowing detailed description thereof taken in connection with theseveral figures of the accompanying drawings, which form a part of thisapplication.

FIG. 1 shows an example of a presently used navigation system thatsupports minimally invasive surgery.

FIG. 2 shows a mock brain and the output of a navigation system showingcorrect registration of a tool.

FIG. 3 shows a mock brain and the output of a navigation system showingmis-registration of a tool.

FIG. 4 shows a navigation system diagram including fiducial touchpointsused for registration of the mock head and brain.

FIG. 5 shows a navigation system diagram showing correct registration ina mock surgery, of an overlay of the tracked virtual objects and theiractual counterparts in real time and actual space.

FIG. 6 shows a mock brain and the output of a navigation system showingincorrect registration of the mock brain.

FIG. 7 shows a diagram of the registration process wherein the virtualmock brain is registered with its actual counterpart using touchpointsand fiducials.

FIG. 8 shows a diagram of the registration process wherein the virtualmock brain is registered with its actual counterpart using patternrecognition.

FIG. 9 shows two views of outputs of the navigation system with correctregistration.

FIG. 10 shows a diagram of a mock brain, a mock head, a mock headholder, and tracking marker reference affixed to the mock head holder ina known and fixed location.

FIG. 11 shows a diagram of both the “camera fixed” and “reference fixed”modes of the navigation system.

FIG. 12 shows a diagram of the navigation system showing theregistration being corrected.

FIG. 13 shows a flow chart depicting the averaging of registration whenusing multiple reference markers.

FIG. 14 shows an automatic re-registration of a virtual tracked toolwith its actual counterpart.

FIG. 15 shows a flow chart showing the steps involved in a port-basedbrain surgery procedure including monitoring of registration.

FIG. 16 shows a flow chart describing two possible registration methods.

FIG. 17 shows a navigation system screen shot of a mock medicalprocedure showing real objects overlaid by virtual objects.

FIG. 18 (a) shows a diagram of a medical pointer tool with an opticaltracking assembly.

FIG. 18 (b) shows a diagram of a medical pointer tool with a templateand optical tracking assembly.

FIG. 19 shows a diagram of a port-based neurosurgery.

FIG. 20 shows a flow chart describing automatic detection andreregistration of a misaligned tracked object.

FIG. 21 shows a diagram of a removable tracking reference marker.

DETAILED DESCRIPTION

Various apparatuses or processes will be described below to provideexamples of embodiments of the disclosure. No embodiment described belowlimits any claimed disclosure and any claimed disclosure may coverprocesses or apparatuses that differ from those described below. Theclaimed disclosures are not limited to apparatuses or processes havingall of the features of any one apparatus or process described below orto features common to multiple or all of the apparatuses or processesdescribed below. It is possible that an apparatus or process describedbelow is not an embodiment of any claimed disclosure.

Furthermore, numerous specific details are set forth in order to providea thorough understanding of the embodiments described herein. However,understood is that the embodiments described herein may be practicedwithout these specific details. In other instances, well-known methods,procedures and components have not been described in detail so as not toobscure the embodiments described herein. Also, the description is notto be considered as limiting the scope of the embodiments describedherein.

Furthermore, in the following passages, different aspects of theembodiments are defined in more detail. In particular, any featureindicated as being preferred or advantageous may be combined with atleast one other feature or features indicated as being preferred oradvantageous.

As used herein, the phrase “intra-operatively acquired imaging” refersto images of a medical procedure being performed on an anatomical part.The imaging procedure may include using a sensor to acquire a continuousintra-operatively acquired image stream (i.e. obtained for example by avideo camera corresponding to real-time imaging) or an intra-operativelyacquired image taken at one or more specific times during the procedureusing a sensor other than a video camera, for example a sensor (such as,but not limited to a camera) which is configured to record specifictypes of images one by one. The present disclosure includes bothmodalities.

In an embodiment, there is provided a continuously available andreal-time confirmation of registration, with an intuitive interface forverification and correction (if necessary). In the embodiment, anoverlay of computed tracked instrument visualization and patient imaginginformation on a video image of the surgical field is provided during aprocedure. In FIG. 2 a surgical tool and its virtual representation areshown aligned (210). In this image, any registration errors may be seenand recognized by an observer (such as a surgeon), as a simplemisalignment of the computed tracked instrument visualization and theactual physical object seen on the video image. An example of erroneousregistration (or mis-registration) is shown in FIG. 3 in which thevirtual representation 310 of the surgical tool is seen to be displacedfrom the actual surgical tool by a distance 320. The surgicalinstrument(s) may be tracked with one or more sensors which are incommunication with one or more transceiver(s) of the tracking systemsthat receive, record and/or process the information regarding theinstrument(s) that the sensor(s) are detecting. The sensors may track,among other things, the spatial position of the instrument(s), includingits angle and orientation, i.e., pose.

Appreciated is that being able to visualize a medical instrument when itis within a patient will aid in the improvement of the accuracy of theprocedure. This can be seen in FIG. 17 as the instrument may bevisualized through the image of a mock patient.

Referring to FIG. 7, in an embodiment a coordinate frame (730) of thenavigation system may be spatially registered with the coordinate frame(740) of the patient imaging data through the respective alignment ofcorresponding pairs of virtual and actual points as described below.This procedure results in the formation of a common coordinate frame. Inan embodiment shown in FIG. 4 the virtual space markers (420) can beseen on a mock patient's pre-operative MR scan, while the actual spacefiducials (1000) corresponding to the virtual space markers can be seenin FIG. 10 on the head of the mock patient. Referring again to FIG. 7there is shown an embodiment of the registration process in which avirtual camera (710) is aligned with the actual camera (720) for thepurposes of rendering a 3D scan of the mock brain such that the renderedposition, orientation and size match the image from the actual cameraand is overlaid onto the real-time video stream of the surgical area ofinterest. A virtual camera is the point and orientation in virtual 3Dspace that is used as a vantage point from which the entire virtual 3Dscene is rendered. The virtual markers are objects within this virtual3D scene. It is understood that those familiar in the art will recognizethis methodology as a standard computer graphics technique forgenerating virtual 3D scenes.

The present system may be used with any compatible surgical navigationsystem. A non-limiting example of such a surgical navigation system isoutlined in the co-pending U.S. patent application Ser. No. 10/433,763,entitled “SYSTEMS AND METHODS FOR NAVIGATION AND SIMULATION OF MINIMALLYINVASIVE THERAPY”, United States Patent Publication US20150351860 basedon U.S. patent application Ser. No. 14/655,814, which claims thepriority benefit of U.S. Provisional Patent Application Ser. Nos.61/800,155 and 61/924,993, wherein for the purposes of this presentUnited States Patent Application, the Detailed Description, and Figuresof United States Patent Publication US20150351860, all of which arehereby incorporated by reference in their entirety.

In an embodiment, the validation system is used in a port-based surgerythe phases of which are depicted in FIG. 15. This flow chart illustratesthe steps involved in a port-based surgical procedure using a navigationsystem. The first step in this surgical procedure involves importing theport-based surgical plan (step 1505) into the navigation system. Adetailed description of a process to create and select a surgical planis outlined in the co-pending “SYSTEM AND METHOD FOR PLANNING,NAVIGATION DURING INVASIVE THERAPY” United States Patent PublicationUS20160070436 based on U.S. patent application Ser. No. 14/769,668,which claims the priority benefit of U.S. Provisional Patent ApplicationSer. Nos. 61/800,155 and 61/924,993, and wherein for the purposes ofthis present United States Patent Application, the Detailed Description,claims and Figures of United States Patent Publication US20160070436,all of which are hereby incorporated by reference in their entirety.

Once the plan has been imported into the navigation system (step 1505),the anatomical part of the patient is affixed into position using a heador body holding mechanism. The patient position is also confirmed withthe patient plan using the navigation software (step 1510). In thisembodiment, the plan is reviewed and the patient positioning isconfirmed to be consistent with craniotomy needs. Furthermore, aprocedure trajectory may be selected from a list of planned trajectoriesproduced in the planning procedure.

Returning to FIG. 15, the next step is to initiate registration of thepatient (step 1515). The phrase “registration” or “image registration”refers to the process of transforming different sets of data into aunitary coordinate system. Appreciated is that there are numerousregistration techniques available and one or more of them may be used inthe present application such as the one described in this disclosure andshown in FIG. 7. Non-limiting examples include intensity-based methodswhich compare intensity patterns between images via correlation metrics,while feature-based methods find correspondence between images usingfeatures such as points, lines, and contours as further described belowand as shown in FIG. 8. Image registration algorithms may also beclassified according to the transformation models they use to relate thetarget image space to the reference image space.

FIG. 16 is a flow chart illustrating the further processing stepsinvolved in registration as outlined in FIG. 15. In this exemplaryembodiment, registration can be completed using a fiducial touchpointmethod (1680) in which the location of touchpoints are registered usinga pointing tool. The fiducial touchpoint method (1680) is describedfurther below and is shown in FIG. 7. Two non-limiting examples ofpointing tools used in a fiducial touchpoint method are shown in FIG.18. Pointing tool (1820) is shown having a template, and pointing tool(1810) is shown without a template, as further described below.

Registration can also be completed by conducting a surface scanprocedure shown generally at (1690) in FIG. 16. Typically for anavigated brain surgical procedure the first step (1620) of a surfacescan involves scanning the face using a laser or other scanning device.The next step is to extract the face surface from MR/CT data (step1640). Finally, registration is performed using a surface-matchingmethod.

As described herein, the overlay of registered virtual and correspondingreal objects in the surgical suite displayed by the navigation systemallows for the identification of mis-registration arising between avirtual object and its corresponding real object in the surgical suite.As shown in FIG. 15 in each step of the procedure following confirmationof registration step (1520) the user (surgeon or other member of thesurgical team) may readily determine the degree of accuracy ofregistration, as the user is permitted to confirm registration in step(1525) of the virtual and actual objects in the OR before moving forwardto the next step in the medical procedure. However, if at any time adiscrepancy, for example as seen at (320) in FIG. 3, between the virtualoverlaid object and actual object is recognized the system may bere-registered (1590) using one of the embodiments described herein.

The subsequent steps after initial registration (1515) and confirmationof registration (1520) in a port-based procedure are further outlined inFIG. 15 and are further described below. Initial registration typicallyhas occurred before the patent is draped (1530). Draping typicallyinvolves covering the patient and surrounding areas with a sterilebarrier to create and maintain a sterile field during the surgicalprocedure. The purpose of draping is to eliminate the passage ofmicroorganisms (i.e., bacteria) between non-sterile and sterile areas.

Upon completion of draping (step 1530), the next step is to confirmpatient engagement points (step 1535) and then prepare and plan thecraniotomy (step 1540). Upon completion of the prep and planning of thecraniotomy step (step 1540), the next step is to cut craniotomy (step1545) where a bone flap may be removed from the skull to access thebrain. The above steps of draping, and performing craniotomy, are knownin the art to add to registration inaccuracy. The next step is toconfirm the engagement point and the motion range of the port (step1555), and once this is confirmed the procedure typically advances tothe next step of cutting the dura at the engagement point andidentifying the sulcus (step 1560).

Thereafter, the cannulation process may be initiated (step 1562).Cannulation involves inserting a port into the brain, typically along asulcus path as identified in step 1560 using an obturator (introducer).Cannulation may be an iterative process that involves repeating thesteps of aligning the port, setting the planned trajectory (step 1580),and then cannulating to a target depth (step 1585), until the completetrajectory plan is executed (step 1562). The surgery then proceeds (step1565) by removing the obturator (introducer) from the port allowingaccess to the surgical site of interest. The surgeon then performtreatment at the distal end of the port, which may involve resection(step 1570) to remove part of the brain and/or tumor of interest.Lastly, the surgeon typically removes the port, close the dura and closethe craniotomy (step 1575).

FIG. 19 is an illustration showing tracked tools in a port-basedsurgical procedure. In FIG. 19, surgeon (1910) is resecting a tumor inthe brain of a patient (1960), through port (1950). External scope(1920), attached to mechanical arm (1930), is typically used by thesurgeon to enhance visibility of the brain at the distal end of the port(1950). The external scope (1920) may be zoomed-in or zoomed-out, andits output depicted on a visual display (not shown) which may beoverlayed with the virtual image of the actual objects contained in thefield of view of the external scope (1920), allowing for the validationof registration as described herein.

Active or passive fiduciary markers (1020) may be placed on the port(1950) and/or imaging sensor (same as 720 in FIG. 7) to determine thelocation of these tools by the tracking camera (1110) and navigationsystem. These markers (1020) may be reflective spheres configured to beseen by the stereo camera of the tracking system to provide identifiablepoints for tracking. A tracked instrument in the tracking system istypically defined by a grouping of markers (1830), which identify avolume in the tracking system, and are used to determine the spatialposition and pose of the volume of the tracked instrument in threedimensions. Typically, in exemplary tracking systems, a minimum of threespheres are required on a tracked tool to define the instrument;however, using four markers (1830) is preferred. Markers (1020) may bearranged statically on the target on the outside of the patient's bodyor connected thereto. Tracking data of the markers acquired by thestereo camera are then logged and tracked by the tracking system. Anadvantageous feature is the selection of markers that can be segmentedeasily by the tracking system against background signals. For example,infrared (IR)-reflecting markers (1020) and an IR light source from thedirection of the stereo camera can be used. Such tracking system, forexample, is the “Polaris” system available from Northern Digital Inc.

In a preferred embodiment, the navigation system may utilize reflectivesphere markers in combination with a stereo camera system, to determinespatial positioning and pose of the medical instruments and otherobjects within the operating theater. Differentiation of the types ofmedical instruments and other objects and their corresponding virtualgeometric volumes could be determined by the specific orientation of thereflective spheres relative to one another giving each virtual object anindividual identity within the navigation system. This allows thenavigation system to identify the medical instrument or other object andits corresponding virtual overlay representation, i.e., the correctoverlay volume, as seen as (310) in FIG. 3. The location of the markersalso provides other useful information to the tracking system, such asthe medical instrument's central point, the medical instrument's centralaxis and orientation, and other information related to the medicalinstrument. The virtual overlay representation of the medical instrumentmay also be determinable from a database of medical instruments.

Other types of markers that could be used would be RF, EM, LED (pulsedand un-pulsed), glass spheres, reflective stickers, unique structuresand patterns, where the RF and EM would have specific signatures for thespecific tools they would be attached to. The reflective stickers,structures and patterns, glass spheres, LEDs could all be detected usingoptical detectors, while RF and EM could be picked up using antennas.Advantages to using EM and RF tags would include removal of theline-of-sight condition during the operation, whereas using anoptical-based tracking system removes the additional noise anddistortion from environmental influences inherent to electrical emissionand detection systems.

In a further embodiment, 3-D design markers could be used for detectionby an auxiliary camera and/or optical imaging system. Such markers couldalso be used as a calibration pattern to provide distance information(3D) to the optical detector. These identification markers may includedesigns such as concentric circles with different ring spacing, and/ordifferent types of bar codes. Furthermore, in addition to using markers,the contours of known objects (i.e., side of the port) could be maderecognizable by the optical imaging devices through the tracking system.

In another further embodiment, the medical instrument may be made orconfigured with an additional protrusion or feature that would notnormally be obscured during a procedure, so that such protrusion orfeature would typically be visible to the optical sensor during theprocedure. Having such feature or protrusion on a tool would enable averification of registration despite the fact that other portions of thetool may be obscured by patient anatomy, or other objects. As such, insuch an embodiment it would be possible to verify registration withouthaving to remove the tool from the patient.

Referring to FIG. 7, in use of the navigation system, its trackingsystem will provide the navigation system a coordinate frame (740),containing the actual spatial locations of the tracked elements in theoperating room, and their spatial relation to one another. Examples ofsuch tracked elements would be the surgical real-time imaging camera(720), which may be a moveable camera used for visualization of thesurgical area of interest, a surgical volume of interest such as abrain, and/or medical instruments. A 3D virtual volume representingpre-operative image data (510) (shown in FIG. 5) of patient anatomy isalso provided to the navigation system.

In an embodiment, the virtual volume is acquired using a patient withattached fiducials (1000, as shown in FIG. 10). The fiducials (1000)remain attached in place on the patient (or else their locations havebeen marked on the patient (as shown in FIG. 4 at (410)) in a mannerwhich persists through the registration step) in order to register thepre-operative imaging data with the patient in the operating room. Thisis illustrated in FIG. 4, which shows two virtual fiducial markers (430)within a mock patient's head scan being in the same position relative tothe virtual mock brain having the actual fiducial markers (1000) as seenin FIG. 10). The spatial correspondence between the actual fiducials andthe virtual fiducials permits their respective coordinate frames to bealigned, which allows for an accurate overlay of virtual image data ontothe actual image data. The overlay is achieved by combining video from avirtual camera (710) (shown in FIG. 7) depicting the virtual operatingroom (OR) surgical field and video from an actual surgical imagingcamera (720) (FIG. 7) depicting the actual OR surgical field. To obtainan accurate overlay, the two cameras (710 and 720) must becoincidentally aligned and have the same optical properties. Hence, thealignment of virtual camera (710) in the navigation system coordinateframe (730) is constrained to be equivalent to the alignment of theactual camera (720), relative to operating room coordinate frame (740),and have the same optical properties as the actual camera (720), namely,the same field-of-view, aspect ratio, and optical distance. This isaccomplished using the navigation system. Given an initial discrepancyor spatial separation (715) between coordinate frames (730 and 740), atracked pointer (748) controlled by a user (750) can be used to confirmthe spatial location of the actual fiducials in virtual space asdepicted in picture frame (780) in the upper right hand side in FIG. 7.

In general, each time a point is identified, the virtual and actualcoordinate frames, (730) and (740) respectively, become more accuratelyaligned. For example, as the tip of the pointer (748) in FIG. 7indicates the spatial position of a fiducial in actual space (locatedabove the left eyebrow of the mock patient), its virtual counterpartfiducial aligns with it resulting in the navigation system coordinateframe (730) to transform (760) and align its origin with the operatingroom coordinate frame (740). This also results in the two cameras (710)and (720) realigning themselves accordingly. The relative shift inalignment of cameras (710) and (720), shown between diagrams (790) and(700), is proportional to the shift between the virtual alignment of theoverlay on the actual image data between (790) and (700). However, giventhat the coordinate frames (730) and (740) respectively are stillrotationally misaligned, as can be seen in the bottom left picture frame(700) in FIG. 7, the alignment process is repeated and another point isregistered. In this iteration the fiducial being aligned is located nearthe right ear of the mock patient and this causes a rotation of thecoordinate frame (730) resulting in it and the coordinate frame (740) tobetter coincidentally align. Repetition of the above steps results inthe production of a common coordinate frame, and accurate registration,as can be seen in diagram (705) (in the lower right hand side of FIG. 7)which shows the accurate overlay of the virtual and mock brain as aresult of the coincident alignment of the virtual and actual cameras(710) and (720), respectively.

As shown in FIG. 11, a tracking camera 1140 (i.e., an opticalmeasurement system that measure the 3D positions of either active orpassive markers or landmarks) is placed in a known position relative toan immobilized patient. As shown in FIG. 5, a computed image (510) frommock patient image data that has been previously obtained or is beingobtained (intra-operatively, including by way of x-ray, MRI, CT,ultrasound, and/or PET, among other modalities) and/or the virtualrepresentation of a medical instrument (210), may then be overlaid ontothe video image, as shown in picture frame (530). A video sensor (1110)as seen in FIG. 11 is a camera used for visualization of the surgicalarea of interest (for example which may be an external scope, or widefield camera), whereas tracking camera (1140) is a 3D tracking sensor (anon-limiting example being a “Polaris Spectra” camera).

In an embodiment, any tracked medical instrument(s) and 3D MR image datais computed for display as an overlay in the live video image feed,positioned relative to the registration transform, (for example a dataoverlay corresponding to an anatomical part (510), and the anatomicalpart). This would show alignment of the computed display with the videoimage of both the instrument(s) and the contours of the anatomical dataif registration is correct, as shown in the bottom image (530) of FIG.5. Any misalignment between the overlay and the actual video image ofthe tool would be immediately noticeable as a mis-registration, andindicate an error in the tracking system registration. This can be seenas the displacement (320) between the projected (overlaid) toolrepresentation and the actual tool in FIG. 3 discussed above. Further,any misalignment between the overlay and the actual video image of ananatomical part would be immediately noticeable as a mis-alignment, andindicate either an error in the tracking system registration or adeformation of the anatomical tissue relative to the previously acquireddataset, either or both of which are useful information for the surgeonand indicates some form of misalignment or non-concordance which shouldbe taken into account and a correction considered.

In an embodiment, a surface rendering of the MR, CT, ultrasound or othermedical imaging data can be generated and displayed in a way to matchthe viewing position and optical properties (e.g. such as zoom, field ofview, etc.) of the viewing camera. As this rendering is dependent on thecomputed registration between the image (or MR, CT, ultrasound or othermedical imaging) dataset and the physical camera position, anymis-registration will tend to be instantly visible as a misalignment inthe overlay display, and can be used to dynamically validate and ensureconfidence in the current registration. An example of a misalignment inthe overlay display can be seen in FIG. 6.

Further, if a mis-registration is detected, a registration correctioncan be applied by manipulating the rendering of the MR, CT, ultrasoundor other medical imaging data on the screen (for example, by rotation,translation, scaling, and any combination thereof) until it matches theoverlaid video, or expressed more generally, until the virtual objectsin the rendering are aligned with the real objects in theintra-operative imaging. In addition to rotation, translation, andscaling corrections, above, the rendering may also be skewed ormanipulated non-linearly (such as optical flow) to generate an alignmentwith the real objects. Examples of linear translation, and rotation, areshown in the diagram in FIG. 12, moving from 1210 to 1220. Thismanipulation may be done automatically when the computer processor isprogrammed with instructions/algorithms to perform these manipulationsand can detect when the virtual and real objects are aligned.

Alternatively, the user/clinician at the surgical workstation may,through a user interface connected to the computer processer, manipulatethe virtual objects manually to align them with their real counterpartsin the intra-operative real time image. The applied manipulation used toachieve the coherent alignment of the virtual and real objects in theimaging data can then be used to compute an update to the registration,which may then be carried over to the overlay of the computed image ofthe instrument(s) from the tracking system. An example of this is shownin FIG. 14. The update to the registration of the instrument(s) isaccomplished by applying the same spatial transform that was applied tothe imaging data to the tracked instrument(s). This may be useful inapplications where it is not desirable or possible to adjust or move apatient to achieve registration correction. In the embodiment, thecorrection to registration is calculated and then can be appliedglobally to imaging, tracking and display systems in order to validateand re-register, without any need to move or reposition a patient.Appreciated is that this process may be automated as well as manuallyapplied.

An example embodiment of an automatic misalignment detection andcorrection process is shown in FIG. 20. In this embodiment, assumed isthat the virtual volumes obtainable using the optical tracking markers,and/or the template (1840), will be the same volume and that they areobtained from a database contained within the navigation system. Alsoassumed is that the coordinates of the volumes (x_(o), y_(o), z_(o),α_(o), β_(o), γ_(o)) and (x_(t), y_(t), z_(t), α_(t), β_(t), γ_(t)),respectively are located in the same location relative to the respectivevirtual volumes, i.e., the virtual optical tracking marker and templateare located relative to the virtual volume in the same location as thereal optical tracking markers and template are located relative to thereal volume. The flow chart depicted in FIG. 20 providing an exemplaryprocess for misalignment detection and correction is explained below inmore detail.

The first step (2000) in the process is to identify and locate theTracking Reference Marker of the Object of interest (TRMO) (for example,(1010) shown in FIG. 10 or (1830) shown in FIG. 18(a)) using a trackingdevice (such as (1140) shown in FIG. 11). The following step (2005) isto obtain the Virtual Volume of the Object of Interest (VVOR) (such asthe virtual volume (310) corresponding to the object of interest (320)shown in FIG. 3) and its spatial position and pose relative to and basedon the identity of the TRMO (as described above), to be overlaid on theimaging feed of the imaging sensor (for example (720) as shown in FIG.7).

Step (2010) is to register the VVOR location in the common coordinateframe by assigning it a coordinate value describing its exact locationand pose in the common coordinate frame relative to the coordinates ofthe TRMO, as assigned below for example:(x_(o),y_(o),z_(o),α_(o),β_(o),γ_(o)), wherein the subscript “o” denotesthe coordinates of the virtual volume of the object of interest asdetermined by the tracking device.

The following step (2015) is to identify and locate the TrackingReference Marker of the Imaging Sensor (TRMS) using the tracking device.Step (2020) is to register the TRMS location in the common coordinateframe by assigning it a coordinate value describing its exact locationand pose in the common coordinate frame as assigned below for example:(x_(s), y_(s), z_(s), α_(s), β_(s), γ_(s)), wherein the subscript “s”denotes the coordinates of the imaging sensor in the common coordinateframe.

The next step (2025) is to obtain the imaging feed acquired from theimaging sensor using the navigation system. The next step (2030) is toalign the virtual imaging sensor with the imaging sensor in the commoncoordinate frame using the TRMS (i.e. so that the views of the twocameras are aligned as shown in the bottom right frame (705) of FIG. 7,as represented by the coincident alignment of both imaging sensors (710)and (720). The following step (2035) is to overlay the VVOR onto itsreal counterpart object in the imaging feed via the common coordinateframe as described in this disclosure.

Step (2040) is to utilize a template matching technique to determine theidentity, location, and orientation of the object of interest, relativeto both the coincidentally aligned virtual and actual imaging sensors((710) and (720) respectively) by detecting the Template Located on theObject of Interest (TLOI) (for example template (1840) attached toobject of interest (1820) shown in FIG. 18(b)). Such template matchingtechniques are known, examples of which are described in the paper[Monocular Model-Based 3D Tracking of Rigid Objects: A Survey; Lepetitet al. published 30 Aug. 2005; Foundations and Trends® in ComputerGraphics and Vision]. Other 3D tracking methods can be used to determinethe exact location, orientation, and identity of the object of interest,also as described in the paper [Monocular Model-Based 3D Tracking ofRigid Objects: A Survey].

The next step (2045) is to obtain the virtual volume of the object ofinterest (VVOT) and its orientation relative to and based on theidentity of the TLOI. The next step (2050) once given the object'slocation and orientation (its spatial position and pose) according tothe TLOI relative to the imaging sensor, is to assign the VVOT of theobject a coordinate value describing its exact location and orientationin the common coordinate frame relative to the coordinates of the TLOIas shown below for example: (x_(t), y_(t), z_(t), α_(t), β_(t), γ_(t)),wherein the subscript “t” denotes the coordinates of the virtual volumeof the object of interest as determined by the imaging sensor.

Step (2055) is to subtract the coordinates of the VVOT and VVOR as shownbelow for example: (x_(q), y_(q), z_(q), α_(q), β_(q),γ_(q))=(x_(o),y_(o),z_(o),α_(o),β_(o),γ_(o))−(x_(t), y_(t), z_(t),α_(t), β_(t), γ_(t)), wherein the subscript “q” denotes the deviation inlocation and orientation (spatial positioning and pose, respectively) ofthe overlaid and real objects in the imaging feed (for example (320) and(310) as shown in FIG. 3), and thus defines the test coordinate. This“test coordinate” is to be used as a test metric to determine the extentof misalignment.

Step (2060) is to compare the obtained test coordinate in the prior stepto a threshold metric to determine if the extent of misalignment of theoverlaid and real objects in the imaging feed as well as the commoncoordinate frame exceed a threshold, for example, as follows:x_(q)>x_(Q); y_(q)>y_(Q); z_(q)>z_(Q); α_(q)>α_(Q); β_(q)>β_(Q); or; andγ_(q)>γ_(Q), wherein the subscript “Q” denotes the coordinates of athreshold metric used to determine if the virtual and real objects ofinterest are misaligned outside of a given tolerance, termed the“threshold coordinate”.

The next step (2065), if the test coordinate is greater than thethreshold coordinate, is to convert the test coordinate obtained in step(2055) into a translation transform and apply it to the VVOT to assignit a new location relative to the TRMO in the common coordinate frame,as follows, for example: (x_(oa), y_(oa), z_(oa), α_(oa), β_(oa),γ_(oa))=(x_(o), y_(o), z_(o), α_(o), β_(o), γ_(o))−(x_(q), y_(q), z_(q),α_(q), β_(q), γ_(q)), wherein the subscript “oa” denotes the coordinatesof the overlaid virtual volume (VVOR) correction.

This step (2055) also entails then setting the newly obtained VVORcoordinate to complete the correction, as follows, for example: (x_(o),y_(o), z_(o), α_(o), β_(o), γ_(o))=(x_(oa), y_(oa), z_(oa),α_(oa),β_(oa), γ_(oa)). The next step is step (2060), if the testcoordinate is less than the threshold coordinate, or, if step (2055) iscompleted, then the next step is returning to step (2000) and restartingthe loop.

In a further embodiment, the system can add registration using videooverlay-match for MR, CT, ultrasound and any other imaging modality,with the addition of annotations of features on the image, (for examplewhich may include solid outlines covering the port opening contour).These overlays can be fixed while the underlying medical imagerepresentation is manipulated (such as for a registration correction). Aregistration is achieved by manipulating the underlying MR to matchthese overlay positions, such as in the dimensions of the image data setor in three-dimensions. For example, three-dimension data points fromtracked instrument(s), and patient features (such as tip of nose, cornerof eyes, edge of ears, positions of bony protrusions, vesselbifurcations, etc.) may be overlaid, or the system can utilize landmarkssuch as a drawing of a surface of the patient or tracing structure (e.g.sulci, ears, exposed vessels) through a tool and the tracking system.

An example of this may be seen depicted in the upper picture frame (810)in FIG. 8 as the inter-hemispherical fissures (820 and 830) of therendered virtual and actual brains, respectively, are overlaid In otherembodiments, overlays can be processed from the MR, CT, ultrasound orother medical images, so that they rotate with the manipulation of themedical image data, and a registration is achieved when the overlay ispositioned to match the corresponding visible tissue location, forexample, such as through segmentation of vessels, segmentation of tumor,skull surface tessellation, automatic detection of facial features (suchas the tip or contour of nose, ears, etc.).

In one embodiment, tracking of tools may be employed to improve therendering of the optical images. A first improvement may be obtained bymasking out the upper (relative to the bottom of the access port)portion of the inserted tools/surgical instruments. Often the tools areout of focus in the optical field at the top portion of their locationinto the access port. Here the image often experiences glare or out offocus issues. Since the system can track the tools and register thetools with a video image, a portion of the tool may be masked out.Masking may be performed, for example, based on known geometrical modelsof the tools, and/or based on real-time image segmentation as describedherein as well as in the paper noted above, entitled “MonocularModel-Based 3D Tracking of Rigid Objects: A Survey.” For example, theupper portion of the tools, or another pre-defined portion of the tools,may be masked or otherwise modified. Accordingly, image artifacts may bereduced, and the ability to utilize the entire dynamic range of thesystem may be improved or enabled.

Additionally, in a related embodiment, the system may be employed toreplace the selected region of the tracked tool with a rendered versionof the tool that follows the three-dimensional profile of the tool,optionally including rendered features such as a shadowed rendering thatindicates and/or emphasizes the change in the diameter of the tool, asit is further distal in the access port. This provides an opportunity toenhance three-dimensional understanding of tool locations. The toolwould then be represented with a partial real-time video view of theactual tip, and a computer rendered view of the upper portion of thetool.

By focusing the camera's gaze on the surgical area of interest, aregistration update can be manipulated to ensure the best match for thatregion, while ignoring any non-uniform tissue deformation areasaffecting areas outside the surgical field of interest. By way ofexample, by focusing the imaging sensor (720) on the surgical area ofinterest, a re-registration can be configured to ensure the best matchfor that particular region (as shown as (850) in the lower picture framein FIG. 8), while ignoring any non-uniform tissue deformation areasoutside such area of interest (840) as shown as (860) outside of thesurgical area of interest (850) in FIG. 8. This can be particularlyuseful in tissue areas that have undergone small morphological changes,such as through swelling after a craniotomy opening. In these cases, themisalignment may not be due to a mis-registration, but primarily due totissue deformation.

Additionally, by matching overlay representations of tissue with anactual view of the tissue of interest, the particular tissuerepresentation can be matched to the video image, and thus ensuringregistration of the tissue of interest (850). For example, theembodiment can: match video of post craniotomy brain (i.e. brainexposed) with imaged sulcal map as shown in FIG. 8; match video positionof exposed vessels with image segmentation of vessels; and/or matchvideo position of lesion or tumor with image segmentation of tumor;and/or match video image from endoscopy up the nasal cavity with a bonerendering of bone surface on nasal cavity for endonasal alignment.

In other embodiments, multiple cameras (or a single camera moved tomultiple positions) can be used and overlayed with tracked instrument(s)views, and thus allowing multiple views of the imaging data and theircorresponding overlays to be presented. An example of this may be seenin the diagrams in FIG. 9. This may provide even greater confidence in aregistration, or re-registration in more than one dimension/view.

In an embodiment, recovery of loss of registration may also be provided.As described above, during a navigation procedure a handheld instrumentis tracked using a tracking system, and a representation of theinstrument's position and orientation may be provided and displayed asan overlay on a previously acquired or current image (such as athree-dimensional scan) of a patient's anatomy obtained with an imagingdevice or system (such as ultrasound, CT or MRI). To achieve this, aregistration is needed between the coordinate frame of a trackingsystem, the physical location of the patient in space, and coordinateframe of the corresponding image of the patient. In an embodiment, aregistration would be needed between the physical location of thepatient, as well as the corresponding image of the patient and thetracking device (1110).

In an embodiment, and as shown in FIG. 11, this registration may beobtained relative to a tracked reference marker (1010 shown in FIG. 10and FIG. 11), which is placed in a fixed position relative to thepatient anatomy of interest and thus can be used as a fixed referencefor the anatomy. Generally, this can be accomplished by attaching thereference marker (1010) to a patient immobilization frame (1130) (suchas a clamp for skull fixation device in neurosurgery), which itself isrigidly attached to the patient. However, the reference marker (1010)may be held to the frame (1130), for example, by an arm, which mayinadvertently be bumped and accidentally moved, which creates a loss ofregistration. Additionally, since the reference marker (1010) must bepositioned so that it is visible by the navigation hardware (typicallyrequiring line-of-sight for optical tracking, or otherwise within theobservation or communication field of the tracking device (1140)) thistends to position the reference marker (1010) such that it is in theopen and thus more susceptible to accidental interaction and loss ofregistration. In situations of lost registration, a surgical proceduretends to be stopped while a new registration is computed, although thismay not always be possible if, for example, the registration fiducialpoints or patient skin surface are no longer accessible due to theprogression of the surgical procedure, and thus creating a need forre-registration or, in some cases even disabling navigation for theremainder of the procedure.

In an embodiment, there is provided a system and method for the recoveryof lost registration, while avoiding the need to perform a fullre-registration. In the embodiment, provided are mechanisms forestablishing backup reference positions that can be returned to in theevent of a loss of registration. In the embodiment, this is provided byone or more secondary reference marker(s) being provided for navigationregistration.

The one or more secondary reference marker(s) (for example as shown inFIG. 21) may be positioned during initial registration, or at any timeduring the procedure. The secondary reference marker(s) can optionallybe removed while the standard procedure is being performed through useof the primary reference marker. A secondary reference marker that hasbeen removed can be placed back into is original position in case aregistration is lost, in order to re-establish a registration. Forexample, a fixture may be affixed or built into any other surface thatis stationary relative to the patient during the procedure (e.g. thepatient immobilization frame, the surgical table, the patient skinsurface, or directly to the patient's bone) to enable a reference markerto be repeatedly attached and removed with a high degree of precision.This fixture may accommodate one or more secondary reference marker(s)and/or tool(s). These secondary reference markers and/or instruments maybe repeatedly attached and removed from the fixture, as may be neededfor recovery of registration. The need for repeatability of positioningof the secondary reference marker(s) may also be avoided or reduced insome embodiments if multiple fixation devices are used.

For example, a surgical probe with a round shaft may be able to bepositioned uniquely in all but the rotation axis about the shaft. Usingmultiple tracking reference tools, a best fit registration match can bedetermined from multiple secondary tracking reference positions so thatthe missing rotational information can be calculated as the registrationwhich matches all the secondary reference positions. A secondaryreference tool(s) is attached to the fixture (or fixtures) and aregistration transformation to the secondary reference tool(s) isrecorded at any time after initial registration, by transforming theprimary registration to the secondary marker's (or markers') positionand orientation. Alternately stated, a secondary tracking referencetool(s) (shown in FIG. 21) may be attached to the fixture (or fixtures)by way of a clip (2210) for example, and a subsequent registrationdefined relatively to the secondary tracking reference tool(s) isrecorded at any time after initial registration, by transforming thelocation and orientation of the initial registration's primary trackingreference to the secondary tracking reference. In various embodiments, asecondary registration can be computed at the time of the primaryregistration, or at any time during the procedure by using the samemechanisms (such as fiducial touch-points, surface matching, etc.) asthe primary registration, thus not relying on a transformation from theprimary registration but generating an independent registration to thesecondary reference tool(s).

Once the registration is recorded, this secondary reference tool canoptionally be removed. Since the secondary registration marker(s) doesnot need to be in position during the surgical procedure, the secondaryreference marker(s) can be placed so that it is near the patient andoccluding the surgical field. Generally, the secondary reference toolmay be removed after registration, and need only be returned to theknown position of a registration in order to provide recovery of lostregistration of a primary (or other secondary) reference. However, ifone or more of the secondary reference(s) are maintained in positionduring a procedure they can also be used, which tends to improvereference position accuracy by using an average of all referencepositions at all times to improve noise sensitivity in recordingreference position; and/or providing a warning (such as avisual/audible) upon detection that the relative positions of thereferences has significantly changed. This may be used to provide anindication that one reference or both have moved and that theregistration has been compromised (and so is in need for correction).

FIG. 13 is a flow chart describing a non-limiting process to employmultiple reference markers during a surgery from the offset of theregistration step (1515) as shown in FIG. 15. The first step (1300) inthe process is to identify and locate Reference Marker 1 (“RM1”) in theOR coordinate frame using the tracking device navigation system. Oncelocated and identified the next step (1305) in the process is toincorporate RM1 into the common coordinate frame. Example coordinatesbeing (x_(α),y_(α),z_(α)), wherein the subscript “α” denotes that thecoordinate belongs to RM1. Steps (1310) and (1315) are analogous to thetwo previous mentioned steps, only instead of applying the steps to RM1the steps are applied to the second reference marker (“RM2”). Examplecoordinates for RM2 being (x_(β),y_(β),z_(β)), wherein the subscript “β”denotes that the coordinate belong to RM2.

The next step (1320) is to begin registration of an object of interestwith a pre-selected number of registration points in fixed and knownlocations with respect to it. The object of interest is to be overlaidwith its virtual counterpart during the surgical procedure. Wherein theregistration is completed using the touch-point method depicted in FIG.7 and described herein.

Step (1325) is to define the registration point(s) (such as, 410 in FIG.4) (i.e. registration points 1, 2, 3, . . . , n) location(s) relative tothe reference markers (such as 1010, as shown in FIG. 10) in the commoncoordinate frame, during the registration depicted in FIG. 7, i.e.defining registration point(s) n having coordinates relative to RM1 inthe common coordinate frame in the following example: (rx_(αn), ry_(αn),rz_(αn))=(x_(α)+x_(rαn), y_(α)+y_(rαn), z_(α)+z_(rαn)) or, equivalently,(rx_(αn), ry_(αn), rz_(αn))=(x_(α), y_(α), z_(α))+(x_(rαn), y_(rαn),z_(rαn))) and relative to RM2 in the common coordinate frame in thefollowing example: (rx_(βn), ry_(βn), rz_(βn))=(x_(β)+x_(rβn),y_(β)+y_(rβn), z_(β)+z_(rβn)) or, equivalently, (rx_(βn), ry_(βn),rz_(βn))=(x_(β), y_(β), z_(β))+(x_(rβn), y_(rβn), z_(rβn)), wherein theequality (x_(α)+x_(rαn), y_(α)+y_(rαn), z_(α)+z_(rαn))=(x_(β)+x_(rβn),y_(β)+y_(rβn), z_(β)+z_(rβn)) or, equivalently, (rx_(αn), ry_(αn),rz_(αn))=(rx_(βn), ry_(βn), rz_(βn)) is inherently satisfied at the timeof the execution of the touch-point registration method (shown in FIG.7), and wherein the prefix “r” represents the coordinates of theregistration points and the subscripts “αn”, and “βn”, denote the n^(th)registration points' coordinate relative to the common coordinate framecalculated using their positions relative to RM1 and RM2 respectivelyand “rαn”, and “rβn” denote the n^(th) registration points' coordinatesrelative to RM1 and RM2 respectively. The coordinate points (x_(α),y_(α), z_(α)) and (x_(β), y_(β), z_(β)) are dynamic given that they arethe positional coordinate of RM1 and RM2 respectively in real-time (i.e.they are periodically updated by the tracking device with respect totheir current locations in the common coordinate frame).

On the other hand, the coordinate point locations (x_(rβn), y_(rβn),z_(rβn)) and (x_(rβn), y_(rβn), z_(rβn)) are defined relative to theposition of the Reference Markers (RM1 and RM2 respectively) and areconstant and unchanging throughout the surgical procedure until such atime that a touch point registration process is executed again. Afterthe registration is completed, step (1330), step (1335) in the processis to define the location of the registration points (1 to n) as:(Rx_(n), Ry_(n), Rz_(n))=[(x_(α)+x_(rαn), y_(α)+y_(rαn),z_(α)+z_(rαn))+(x_(β)+x_(rβn), y_(β)+y_(rβn), z_(β)+z_(rβn))]/2 or,equivalently, (Rx_(n), Ry_(n),Rz_(n))=(([(x_(α)+x_(rαn))+(x_(β)+x_(rβn))]/2),([(y_(α)+y_(rαn))+(y_(β)+y_(rβn))]/2),([(z_(α)+z_(rαn))+(z_(α)+z_(rβn))]/2))or, equivalently, (Rx_(n), Ry_(n), Rz_(n))=[(rx_(βn), ry_(βn),rz_(βn))+(rx_(αn), ry_(αn), rz_(αn))]/2, wherein the prefix R denotesthe n registration point(s) average coordinates based on their relativelocation of two reference markers (RM1 and RM2). Step (1340) is to usethe calculated point(s) (Rx_(n), Ry_(n), Rz_(n)) (1 to n) to registerthe real object in the common coordinate frame so it can be overlaidwith its virtual counterpart. It should be noted that if the coordinatesof RM1 and RM2 are constant throughout the procedure the equality(x_(α)+x_(rαn), y_(α)+y_(rαn), z_(α)+z_(rαn))=(x_(β)+x_(rβn),y_(β)+y_(rβn), z_(β)+z_(rβn)) will be satisfied, resulting in thefollowing expression: (Rx_(n), Ry_(n), Rz_(n))=(rx_(βn), ry_(βn),rz_(βn))=(rx_(αn), ry_(αn), rz_(αn)).

This implies the relative position of the registration points (1 to n)will remain in an unchanged location (i.e. have constant coordinates) inthe common coordinate frame equivalent to the initial touch-pointregistration coordinates. However if the points RM1 and/or RM2 changethen the equality is broken and an averaged position located at themidpoints of the registration point sets relative to both RM1 (i.e.(rx_(αn), ry_(αn), rz_(αn))) and RM2 (i.e. (rx_(βn), ry_(βn), rz_(βn)))are calculated and used to determine the location of the virtual objectoverlay in the common coordinate frame.

The next three (3) steps in the process involve identifying anypotential shifts of the reference marker locations to the point wherethe registration becomes inaccurate (as defined by a threshold value).Step (1345) indicates that the instantaneous (last updated) registrationpoints relative to RM1 and RM2 must be subtracted and their absolutevalue calculated and defined as the total deviation and denoted with aprefix “t” as in the following example: (tx_(n), ty_(n),tz_(n))=|(rx_(βn), ry_(βn), rz_(βn))−(rx_(αn), ry_(αn), rz_(αn))|.

Once calculated step (1350) indicates the total deviation of theinstantaneous (last updated) registration points relative to RM1 and RM2will be compared to a threshold value defined by the user as in thefollowing example: If tx_(n)<Tx_(n), or ty_(n)<Ty_(n), or tz_(n)<Tz_(n),wherein the prefix “T” indicates the coordinate threshold values, thenthe process continues a loop by initiating step (1360) which is toupdate of the assigned location of RM1 and RM2 (i.e. (x_(α),y_(α),z_(α))and (x_(β),y_(β),z_(β)) respectively) in the common coordinate framereturning followed by returning to step (1335). However, if (tx_(n),ty_(n), tz_(n))>(Tx_(n), Ty_(n), Tz_(n)), then the process moves to step(1355) and indicates to the surgeon that the registration of the objectis inaccurate.

In an embodiment having multiple reference tools, it is possible toinfer which reference tool has moved, by determining the one that hasshifted position most significantly relative to a camera, and so thisreference tool can automatically be dropped from calculation and theprocedure can continue without interruption. In the embodiment, once amoved reference position has stabilized, a new position for thereference can be recorded and it can automatically be returned tofunction as a fixed reference in its new position, again, withoutinterruption.

In use, if at any time during a procedure, a primary reference marker(1010) or instrument is moved and registration is lost (or, in effectconsidered no longer reliable), one or more secondary referencemarker(s) or instruments(s) (that were previously registered can beaffixed to the fixture and the secondary registration can bere-established using the secondary marker(s) or instruments(s). Theprocedure can continue using the secondary reference marker, or thesecondary reference marker(s) or instruments(s) can be used as a fixedpoint to compute an updated registration to the (now) moved position ofthe primary reference, so that the procedure can continue using theupdated primary reference registration. At that point the secondaryreference marker(s) can optionally be removed from the fixation device,since registration using the primary reference marker (1010) would thenbe updated to reflect the moved position of the primary reference markeror instruments(s).

In an embodiment, a secondary reference marker or tool can be a separatetracking tool, or it can be a sterile, tracked surgical hand-piece orpointer, each of which would have been registered to a fixation fixture,which can be (re)attached temporarily to the fixation fixtures tore-establish a lost registration.

In some embodiments, multiple fixation fixtures can be provided andplaced at any convenient position, each capable of holding a secondaryreference marker(s) and/or having a handpiece or pointer initializedlocations. In the embodiment, at any time during the procedure, any oneof these fixtures can be used to re-establish or refine a registrationby re-attaching the marker(s). In an example, secondary markers can beinclusive of both the tracking reference marker (1010) and the trackingsensor (1140).

In an embodiment, the fixation fixture may be a disposable clip thatadheres, straps or screws into place. As noted above, FIG. 10 shows adiagram of a tracking reference to which passive tracking markers (1020)are affixed in a known and fixed location with respect to the anatomicalpart undergoing the medical procedure, in this case a patient's brain.

In an embodiment, a reference position may be provided, which allows atracking camera to be moved dynamically during a procedure to ensure agood line-of-sight with the tracked instruments. Another method ofrecovering registration is based on the ability to temporarily lock thetracking camera in a fixed position relative to the patient. An exampleof an embodiment is shown in FIG. 11 in the right hand frame (1105). Thesystem (and associated control software) can be placed into a “camerafixed” mode, at which time the tracking camera is not allowed to moverelative to the patient. In the “camera fixed” mode, the registration isestablished relative to this fixed camera, regardless of the position ofthe reference marker, and thus in essence making the camera positionitself the fixed reference.

In this embodiment, if the camera position needs adjustment, then thesystem may be placed into a “reference fixed” position, whichestablishes a registration relative to the position of the reference,and thereafter allows the camera to be moved to a new position withoutaffecting the registration. The embodiment can also subsequently returnto the “camera fixed” mode to again provide independence from theregistration marker. An example embodiment of the system in “referencefixed” mode is shown in FIG. 11. In the left hand frame (1100) thecamera (1140) is moved (1150) to a new position while the patient isregistered to the reference frame (1120) of the reference marker (1010).This embodiment can be extended to allow the system to enter a “tool Xfixed” mode (where X represents any one of the available tracked tools),where any tool can be asserted to be remaining fixed in relative to thepatient, and the registration can transfer to that tool X position.Afterward, the procedure can continue with that tool as a reference, orthe system can be subsequently returned to a “reference fixed” mode oncethe primary reference is not moving relative to the patient, and theprocedure continues with the registration transferred to the originalprimary reference tool.

However, in typical use during a procedure, after initial registrationtypically it will be advantageous that neither the tracking systemsensor nor the reference marker be moved relative to the patient. Aregistration that averages a camera-fixed and reference-fixed computedtransformation can be used to provide a more accurate registration insome embodiments. An algorithm for averaging a registration based upontwo reference markers is depicted in FIG. 13.

The system can also permit for “camera fixed” and “reference fixed”modes (or any combination of “tool X fixed” modes) to be enabled at thesame time. In an embodiment, the restriction would be that a systemstate of “camera movable” and “reference movable” cannot be enabledtogether, or stated more generally, there must always be at least onetool remaining in “fixed mode” to act as a fixed patient reference. Inan embodiment, the system can be configured to respond to an automaticsignal sent when the camera mount is unlocked for movement, thusswitching out of camera locked mode automatically when the camera isbeing moved and shifting back when the camera position is again locked.The tracking of movement of a reference marker or tool can be monitoredduring a procedure, and a warning can be displayed or sounded if thereference has been moved relative to any other fixed reference.

In another embodiment, the use of secondary reference marker(s) and atracking camera (1140) can be combined to provide a registrationvalidation, where no fixed reference marker (1010) is required. In theembodiment, a tracking tool may be affixed to a fixation fixture and theregistration is established relative to the tracking tool prior tomoving the tracking sensor. After the tracking sensor is positioned andthe registration is returned to be relative to the tracking sensorposition, the reference tracking tool can then be removed. In theembodiment, the camera may be considered as a virtual secondaryreference tool at the origin of the camera reference frame, which cameratends to serve the same function as the secondary reference marker ortool as described above in other embodiments.

At least some of the elements of the systems described herein may beimplemented by software, or a combination of software and hardware.Elements of the system that are implemented via software may be writtenin a high-level procedural language such as object-oriented programmingor a scripting language. Accordingly, the program code may be written inC, C++, C#, SQL or any other suitable programming language and maycomprise modules or classes, as is known to those skilled inobject-oriented programming. At least some of the elements of the system10 that are implemented via software may be written in assemblylanguage, machine language or firmware as needed. In either case, theprogram code can be stored on a storage media or on a computer readablemedium that is readable by a general or special purpose programmablecomputing device having a processor, an operating system and theassociated hardware and software that is necessary to implement thefunctionality of at least one of the embodiments described herein. Theprogram code, when read by the computing device, configures thecomputing device to operate in a new, specific and predefined manner inorder to perform at least one of the methods described herein.

Furthermore, at least some of the methods described herein are capableof being distributed in a computer program product comprising a computerreadable medium that bears computer usable instructions for execution byone or more processors, to perform aspects of the methods described. Themedium may be provided in various forms such as, but not limited to, oneor more diskettes, compact disks, tapes, chips, USB keys, external harddrives, wire-line transmissions, satellite transmissions, internettransmissions or downloads, magnetic and electronic storage media,digital and analog signals, and the like. The computer useableinstructions may also be in various forms, including compiled andnon-compiled code.

While the applicant's teachings described herein are in conjunction withvarious embodiments for illustrative purposes, it is not intended thatthe applicant's teachings be limited to such embodiments. On thecontrary, the applicant's teachings described and illustrated hereinencompass various alternatives, modifications, and equivalents, withoutdeparting from the embodiments, the general scope of which is defined inthe appended claims. Except to the extent necessary or inherent in theprocesses themselves, no particular order to steps or stages of methodsor processes described in this disclosure is intended or implied. Inmany cases the order of process steps may be varied without changing thepurpose, effect, or import of the methods described.

What is claimed:
 1. A computer-implemented method of detectingregistration error during a medical procedure by way of a registrationerror detection system, the registration error detection systemcomprising a surgical navigation system and a computer control system,the computer control system comprising a processor, the navigationsystem comprising a tracking system, the tracking system comprising animaging system and a tracking device, the imaging system comprising anactual camera having a virtual camera aligned therewith, and theregistration error detection system operable with a display system, thedisplay system comprising a display, the method comprising: acquiring,in real-time, using the actual camera, an intraoperative image of asurgical field having at least one real object, the at least one realobject previously registered with at least one corresponding virtualobject, the actual camera and the at least one real object previouslyregistered, via the tracking system, to a common coordinate framerelative to a tracked reference marker, and the tracked reference markerdisposed in a fixed position relative to a patient anatomy; displayingthe intraoperative image on the display; acquiring, using the virtualcamera, a virtual image having the at least one virtual object in thecommon coordinate frame; overlaying, in real-time during the procedure,the virtual image in relation to the intraoperative image on thedisplay; detecting any misalignment between any one of the at least onereal object of the intraoperative image and the corresponding at leastone virtual object of the virtual image, wherein a presence ofmisalignment is indicative of registration error between the at leastone virtual object and the corresponding at least one real object, andwherein a coordinate frame of the surgical navigation system isspatially registered with a coordinate frame of patient imaging datathrough respective alignment of corresponding pairs of at least onevirtual point and at least one actual point; automatically correctingthe misalignment based on a difference between at least one local tissuecharacteristic and at least one virtual instrument representation at alocation being focused and a live video stream of the surgical field,automatically correcting comprising: nonlinearly manipulating arendering of a patient image and the virtual image to align with theintraoperative imaging; and nonlinearly manipulating comprising usingoptical flow to generate an alignment with the at least one real object,thereby providing an immediate local registration correction; andglobally applying the local registration correction to at least one ofthe imaging system, the tracking system, and the display system, therebyproviding dynamic validation and re-registration, and therebyeliminating any need to reposition a patient.
 2. The method of claim 1,wherein, if a misalignment is detected in the detecting step, furthercomprising: applying at least one of translating, rotating, skewing, andscaling of the at least one virtual object in the common coordinateframe to align the at least one virtual object with the corresponding atleast one real object in the common coordinate frame for re-registeringthe at least one virtual object; and assigning the reregistered at leastone virtual object as the previously registered at least one virtualobject in the common coordinate frame.
 3. The method of claim 2, whereinthe at least one real object comprises an anatomical part undergoing themedical procedure, and wherein the at least one pre-registered virtualobject in the virtual image comprises a pre-operative image of theanatomical part.
 4. The method of claim 2, wherein the at least one realobject comprises a medical instrument, and wherein the at least onepre-registered virtual object in the virtual image comprises a virtualimage of at least one medical instrument.
 5. The method of claim 1,wherein the at least one real object comprises an anatomical partundergoing the medical procedure, wherein the at least onepre-registered virtual objects in the virtual image comprises apre-operative image of the anatomical part, wherein the at least onereal object comprises a medical instrument, and wherein the at least onepreregistered virtual object in the virtual image comprises a virtualimage of at least one medical instrument.
 6. The method of claim 3,wherein the anatomical part comprises a brain.
 7. The method of claim 6,wherein the surgical field comprises at least one landmark, and whereinthe at least one landmark comprises at least one of: a morphologicalfeature intrinsically associated with the brain, a head of the patient,and a face of the patient.
 8. The method of claim 6, wherein thesurgical field comprises at least one landmark, and wherein the at leastone landmark comprises a plurality of fiducial markers disposed in aplurality of fixed and known positions in relation to the brain.
 9. Themethod of claim 8, wherein the plurality of fiducial markers comprisesat least one of a plurality of active fiducial markers and a pluralityof passive fiducial markers.
 10. The method of claim 8, wherein thesurgical field comprises a plurality of landmarks, wherein the pluralityof landmarks comprises at least one of a plurality of morphologicalfeatures intrinsically associated with the anatomical part and aplurality of fiducial markers disposed in a plurality of preselectedpositions in relation to the anatomical part, and wherein the pluralityof fiducial markers are disposed in a field of view of the trackingdevice.
 11. The method of claim 8, wherein the at least one of realobject comprises a surgical port, and wherein the at least onepreregistered virtual object in the virtual image comprises a virtualimage of the surgical port.
 12. The method of claim 10, using theprocessor, programmed with a set of instructions, further comprising:receiving input specifying at least one of translating, rotating,skewing, and scaling the virtual object in the coordinate frame of thesurgical navigation system to align the at least one virtual object withthe corresponding at least one real object in the coordinate frame ofthe surgical navigation system; re-registering the at least one virtualobject, thereby providing at least one re-registered virtual object;storing the at least one re-registered virtual object in the coordinateframe of the surgical navigation system; and assigning the at least onere-registered virtual object as at least one previously registeredvirtual object.
 13. The method of claim 10, wherein, in the event of amisalignment between the at least one virtual object and the at leastone corresponding real object at a pre-selected time, performing atleast one of translating, rotating, skewing, and scaling of the at leastone virtual object in the coordinate frame of the surgical navigationsystem to align the at least one virtual object with the at least onecorresponding real object in the coordinate frame of the surgicalnavigation system and re-registering the at least one virtual object,and wherein the processor is programmed with instructions to store theat least one re-registered virtual object in the coordinate frame of thesurgical navigation system and assigning the at least one re-registeredvirtual object as the at least one previously registered virtual object.14. The method of claim 1, wherein acquiring the virtual image comprisesusing at least one of CT imaging, MRI imaging, X-Ray imaging, PETimaging, and ultrasound imaging.
 15. A registration error detectionsystem for detecting registration error between at least one previouslyregistered virtual object and an intra-operatively acquired image duringa medical procedure, the registration error detection system comprisinga surgical navigation system and a processor, the navigation systemcomprising a tracking system, the tracking system comprising an imagingsystem and a tracking device, the imaging system comprising an actualcamera having a virtual camera aligned therewith, and the registrationerror detection system operable with a display system, the displaysystem comprising a display, the registration error detection systemcomprising: at least one sensor for acquiring an intra-operative imageof a surgical field during the medical procedure, the surgical fieldcontaining at least one real object corresponding to the at least onepreviously registered virtual object, the processor configured, by a setof instructions, to: acquire, in real-time, using the actual camera, anintraoperative image of a surgical field having at least one realobject, the at least one real object previously registered with at leastone corresponding virtual object, the actual camera and the at least onereal object previously registered, via the tracking system, to a commoncoordinate frame relative to a tracked reference marker, and the trackedreference marker disposed in a fixed position relative to a patientanatomy; display the intraoperative image on the display; acquire, usingthe virtual camera, a virtual image having the at least one virtualobject in the common coordinate frame; overlay, in real-time during theprocedure, the virtual image in relation to the intraoperative image onthe display; detect any misalignment between any one of the at least onereal object of the intraoperative image and the corresponding at leastone virtual object of the virtual image, wherein a presence ofmisalignment is indicative of registration error between the at leastone virtual object and the corresponding at least one real object, andwherein a coordinate frame of the surgical navigation system isspatially registered with a coordinate frame of patient imaging datathrough respective alignment of corresponding pairs of at least onevirtual point and at least one actual point; automatically correct themisalignment based on a difference between at least one local tissuecharacteristic and at least one virtual instrument representation at alocation being focused and a live video stream of the surgical field,by: nonlinearly manipulating a rendering of a patient image and thevirtual image to align with the intraoperative imaging; and nonlinearlymanipulating comprising using optical flow to generate an alignment withthe at least one real object, whereby an immediate local registrationcorrection is provided; and globally apply the local registrationcorrection to at least one of the imaging system, the tracking system,and the display system, whereby dynamic validation and re-registrationis provided, and whereby any need to reposition a patient is eliminated.16. The system of claim 15, wherein the at least one sensor comprises avideo camera configured to record the surgical field in real-time, andwherein the processor is configured to overlay the virtual image of theat least one previously registered virtual object in relation to theintra-operatively acquired image real-time during the medical procedure.17. The system of claim 15, wherein the at least one sensor is a videocamera configured to record the surgical field in real-time, and whereinthe computer processor is configured to overlay the virtual image of theat least one previously registered virtual object in relation to theintra-operatively acquired imaging at preselected times during themedical procedure.
 18. The system of claim 15, wherein the at least onesensor is configured to acquire individual intra-operative images atpre-selected times during the medical procedure, and wherein theprocessor is configured to overlay the virtual image of the at least onepreviously registered virtual object in relation to the intraoperativelyacquired image at the pre-selected times during the medical procedure.19. The system of claim 15, wherein the at least one sensor isconfigured to acquire individual intra-operative images at pre-selectedtimes during the medical procedure, and wherein the processor isconfigured to overlay the virtual image of the at least one previouslyregistered virtual object in relation to the intraoperatively acquiredimage in real-time.
 20. A method of providing a registration errordetection system for detecting registration error between at least onepreviously registered virtual object and an intra-operatively acquiredimage during a medical procedure, the registration error detectionsystem comprising a surgical navigation system and a processor, thenavigation system comprising a tracking system, the tracking systemcomprising an imaging system and a tracking device, the imaging systemcomprising an actual camera having a virtual camera aligned therewith,and the registration error detection system operable with a displaysystem, the display system comprising a display, the method comprising:providing at least one sensor for acquiring an intra-operative image ofa surgical field during the medical procedure, the surgical fieldcontaining at least one real object corresponding to the at least onepreviously registered virtual object, the processor configured, by a setof instructions, to: acquire, in real-time, using the actual camera, anintraoperative image of a surgical field having at least one realobject, the at least one real object previously registered with at leastone corresponding virtual object, the actual camera and the at least onereal object previously registered, via the tracking system, to a commoncoordinate frame relative to a tracked reference marker, and the trackedreference marker disposed in a fixed position relative to a patientanatomy; display the intraoperative image on the display; acquire, usingthe virtual camera, a virtual image having the at least one virtualobject in the common coordinate frame; overlay, in real-time during theprocedure, the virtual image in relation to the intraoperative image onthe display; detect any misalignment between any one of the at least onereal object of the intraoperative image and the corresponding at leastone virtual object of the virtual image, wherein a presence ofmisalignment is indicative of registration error between the at leastone virtual object and the corresponding at least one real object, andwherein a coordinate frame of the surgical navigation system isspatially registered with a coordinate frame of patient imaging datathrough respective alignment of corresponding pairs of at least onevirtual point and at least one actual point; automatically correct themisalignment based on a difference between at least one local tissuecharacteristic and at least one virtual instrument representation at alocation being focused and a live video stream of the surgical field,by: nonlinearly manipulating a rendering of a patient image and thevirtual image to align with the intraoperative imaging; and nonlinearlymanipulating comprising using optical flow to generate an alignment withthe at least one real object, whereby an immediate local registrationcorrection is provided; and globally apply the local registrationcorrection to at least one of the imaging system, the tracking system,and the display system, whereby dynamic validation and re-registrationis provided, and whereby any need to reposition a patient is eliminated.